Improving the Performance of LVCSR Using Ensemble of Acoustic Models

Recent advances in Machine Learning have brought to attention new theories of learning as well as new approaches. Among these, the Ensemble method has received wide attention and has been shown to be a promising method for classification problems. Simply speaking, the ensemble method is a learning algorithm that constructs a set of “weak” classifiers and then combines their predictions to produce a more accurate classification. The underlying idea of the ensemble method is that the combination of diversified classifiers that have uncorrelated, and ideally complementary, error patterns can offer improved performance and a robust generalization capability. Given its successes for many classification problems, we began investigating the problem of adapting ensemble techniques to continuous speech recognition. Continuous Speech Recognition has been acknowledged as one of the most challenging tasks in classification. The performance of an ASR system is negatively impacted by a number of issues, such as corruption of noise, variability of speaker and speaking mode, change of environment conditions, transmission of channel, inaccuracy of model assumption, complexity of language, etc.. The primary goal of our research is to discover methods suitable for ensemble construction and combination that meet these special requirements of continuous speech recognition. We propose several novel ensemble-based acoustic model training and combination schemes, and test their effectiveness using real-world speech corpora. Preliminary results are described in this proposal, in particular • Utterance-level Boosting training algorithm for large scale acoustic modeling • Frame-level Boosting training algorithm using a Word Error Rate reduction criterion • N-Best list re-ranking and Rover combination to generate a better hypothesis Encouraging experimental results convince us that the ensemble technique is a promising method and that it has the potential to substantially improve the performance of a LVCSR system. However research on ensemble methods for speech recognition is still in its early stage and unsolved questions on ensemble generation and hypothesis combination remain to be addressed. This proposal sets out several key research topics that, if successfully addressed will have the potential to significantly increase the accuracy of ensemble-based speech recognition systems. These include the following: • Training criteria targeted at reducing Word Error Rate rather than Sentence Error Rate. • Integrating data manipulation and feature manipulation methods for continuous speech recognition. • Combination methods working on different objects, different levels and different decoding stages. • Ensemble-based semi-supervised acoustic model training algorithm using labeled and unlabeled data.